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bac:k:groxjni> of tbde inveimtioiv 

Field of tlie loventioD 

5 The present invention relates to the apparatus and. method for image configuring 

capiable of cx>nflguring a singLe object image of high resolution firom a plurality of 
reduced object images of low resolution obtained by a compound-eye camera nsing a 
plmnality of micro lenses. 

Oescription of tlie Related Art 

lO As an image input apparatus that effectively obtains various image information 

of high quality, image input apparatuses having structure of obtaining an object image 
by single optical system that is opposing to the obfect as a photogenic subject, such as 
digital cameras and video cameras, have been widely used. However, in these days, 
for the purpose of downsizing and thinning the image input apparatus, compoimd-eye 

1 5 cameras mimicking the compoimd-eye structure of such as insects have been developed. 

A compound-eye camera includes a micro-lens array in which a plurality of 
micro lenses are arranged, as well as a photo detector formed in a planar condition, and 
obtains a plurality of reduced object images of low resolution focused on the photo 
detector by using the micro lens array. Here, when the number arrays of 

20 one-dimensional direction (one column) of the micro lens array is assumed as N, 
compound-eye camera can observe an observation area nearly same as the one for 
single-eye camera at 1/N times of the focal length thickness} of single-eye camera 

by using the photo detector of the same level. This significantly realizes the thinning 
and downsizing of the compoimd-eye camem compared to the single-eye camera 

1 



capable of observing the same observation area, and allows a compound- eye camera to 
obtain a bright image of the same level as tiie one obtained by a single-eye camera. 

Fig. 8 shows an optical system of a single-eye camera ((a) single-eye optical 
system) and an optical system of a compound-eye camera CG^) compomid-eye optical 
5 system) in a general maimer. 

As can be understood by comparing Figs. 8 (a) and (b), since each of the focal 
lengths Qn. Fig. 8, indicated at £^N) of the micro lenses 80 are minimal, a compound-eye 
camera can be structured as a significantly downsized and thin apparatus^ compared to a 
single-eye camera capable of obtaining an image of the same-lervel in brightness. 

10 However, while the compound-eye camera can be thin, small and capable of 

obtaining a bright image, it has a defect that each of the reduced object images is of low 
resolution. Therefore, it is required to configure a single object image of high 
resolution by conducting image processing to these plurality of reduced object images. 

As a method for obtaining a single high-resolution image from a plurality of 

15 reduced object images, for example, a method fbr image reconfiguring by means of 
such as sampling method and pseudo inverse method has been disclosed in J. Tanida, T. 
ICumagai, K. Yamada, S. Miyatake, K. Ishida, T. IVlorimoto, N. Kondo, E). Nliyazaki, 
and VI Ichioka, "Thin Observation module by boimd optics {TOMBO): concept and 
experimental verification^" Appl. Opt, 40, 1806-1813 (2001) (bereinafter referred to as 

20 ''non-patent literature 1"). The sampling method is for reconfiguring a single object 
image by superposing reduced object images basing their centroid positions. The 
pseudo inverse method is firstly for indicating the object as a photogenic subject and the 
reduced object images by vectors, and describing a point spread function of optical 
system by matrix. It then for reconfiguring a single object image by calculating the 
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inverse matrix of tiie point spread function mathematically. 

Additionally, in Japanese Patent Publication No. 2003-141 529 (liereinafter 
referred to as ''patent literature 1"), a method for image reconfiguring that firstly 
calculates a sliift amount of a plurality of reduced object images through correlation 
5 calculation between these reduced object images, and consequently obtains a 
high-resolution image by rearranging each pixel of a plurality of reduced images on a 
same area based on the above shift amount has been disclosed. 

On the other hand, as a digital super-resolution technology based on an image 
obtained by a single-eye camera^ an iterative back projection method as an iterative 
lO algorithm that generates a single high-resolution image firom a plurality of 
low-resolution images having overlaps though having different observation positions 
has been disclosed in M. Irani and S* Peleg, 'Improving resolution by image registration, 
CVGIP: Cjraphical IVIodels and Image Processing", 53, pp. 231-239 C1991) (1aereinaf%er 
referred to as "non-patent literature 2"), 

15 

SUIMQVXARY OF* THE EWENTrON 

However, . a. compound-eye cEunera. is for Imaging a same object firom a 
different point of view by a plurality of minute optical systems, and therefore, a 
plurality of reduced object images include disparities. 
20 Therefore, since the above mentioned sampliag method is a method for merely 

superposing a plurality of reduced object images of low-resolution formed on a photo 
detector, the disparities between reduced object Images become the position fluctuation 
of the object information, and there remains a. problem of failing to achieve 
high-resolution without improvement of the resolution of the reconstructed object 
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image. 

A^ddltionally, in the above mentioned pseudo inverse xnethod, it is theoretically 
X>ossib]e to increase the resolution of reconstructed object image by describing the point 
spread function accurately. However, it is practically di£Scult to obtain an accurate 
5 distance between the object and the micro lens array or between the micro lens array 
and the photo dertector, as well as to accurately obtain the influence firom the 
misalignment of the micro lens array and the photo detector. Therefore, it is 
impossible to accurately describe the above mentioned point spread function. 
Furthermore, in the calculation for inverse matrix, since it is impossible to obtain an 

10 accurate inverse matrix of the point spread function, there is no other than a pseudo 
inverse as an approximate inverse matrix to be utilized. Consequently, there remains a 
problem that it is impossible to avoid lowering of the resolution of the object image 
obtained by reconfiguration using pseudo inverse method. 

Also, according to the pixel rearranging method disclosed in the patent 

1 5 literature 1, there remains a problem that, even though it can obtain an image of higher 
resolution compared with the sampling method and the pseudo inverse method, it 
cannot avoid deteriomtion of image quality caused by the limitation of spatial resolution 
of a compound-eye camera itself compared to the image imaged by a single-eye camera. 

hi a compoimd-eye camera, due to the relation of magnification ratio of optical 

20 system, the observation area per pixel in each reduced object image spreads ISf times (N: 
number of arrays in one column of the micro lens array) of the observation area per 
pixel in a single-eye camera that observes the same observation area, and this indicates 
the spatial resolution per pixel being low (rough). Furthermore, the above mentioned 
pixel reEiiTanging method is basicedly for back-projecting the pixel information on a 
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plxirality of reduced object images onto a reconfiguring surface (the same area) in one 
for one correspondence. This equals to approximately ignoring the opiening spread of 
the photo detector, and therefore, deterioration can be seen in the image quality of the 
single image (reconstructed image) obtained by the pixel rearranging method, compared 
5 with the image imaged by single eye image input appEiratus. For example, when 
analyzing the reconstructed image produced by the pixel rearranging method on 
frequency surface, the result is severely impaired in information of high-frequency 
content. 

Consequently, this invention has been invented considering the foregoing 
lO conditions, and the purpose of this invention is to provide the method and apparatus for 
image configuring capable of configuring a single object image of high resolution from 
a plurality of reduced object images of low resolution focused on a photo detector by a 
micro lens array, and further, capable of configuring a single object image of high 
resolution even when a distance between an object and a micro lens array is inconstant, 
1 5 and when there is a misalignment of a micro lens array and a photo detector. 

In order to achieve the above-mentioned puxpose, the present invention relates 
to an image configuring apparatus constituted as comprising an object image 
configuring means, which reads image data of a plurality of reduced object images from 
a reduced image recording means for recording image data of a plurality of reduced 
20 object images obtained by imaging an object using a compovind-eye camera that focuses 
a plirrality of reduced object Images on a photo detector through micro lens array having 
a plurality of micro lenses arrayed therein, and confrgures a single object image based 
on said image data and then outputs its image data, ATS^erein the object image 
configuring means is characterized by conducting: a generating means of initial object 
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image for generating an initial ixnage data of a single object image based on the image 
data of a plurality of* reduced object images; a reduced image estimating means for 
estimating an estimated image of each of* the reduced object images firom a image data 
of* a provided single object image based on a geometric projection means; an object 
5 image updating means for updating an image data of the single object image provided in 
the reduced image estimating means by projecting a difference between the estimated 
image of each reduced object images and each of the reduced object images in an 
inverse process of the geometric projection process; and an iterative control means for 
firstly giving the initial image data to the reduced image estimating means as an initial 

10 value of the image data of the single object image, and then repetitively conducting the 
reduced image estimating means as well as the object image updating means until the 
dxSerence satisfying the predetermined conditiort, then oiitputting the ixnage data of the 
single object ixnage at the time of the difference satisfying the predetermined condition 
as a final image data of the object image. 

15 This is when the iterative back projection method is applied for the processing 

of configuring a single object image by using a plurality of reduced object images 
obtained by photographing of a compound-eye camera (a compoimd-eye image input 
apparatus). 

Consequently, as described later, it is verified that the present invention can 
20 obtain an object image of the resolution higher than the one of high-resolution image 
obtained by the pixel rearranging method. 

Here, when the above-mentioned object image configuring means conducts a 
shift amoimt calculating pnrocess for calculating a shift: amoimt in regard to a gap of 
relative positions between the reduced object images through correlation calculation 
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between the reduced object images by using the image data of a plurality of the reduced 
object images, the following two applications will be possible by using such shift 
amotxnt. 

The first one is to obtain a conversion equation indicating the geometric 
5 projection process employed in the reduced image estimating process. 

The conversion equation indicating the geometric projection process can be set 
based on such as property of the compound-eye camera and a distance between an 
object and the compound-eye camera. However, since the distance between an object 
and the compoimd-eye camera can change according to the situation, it is not practical 
10 to set an accurate distance in each case. Also, as to the property of the camera, it may 
include a misalignment (the gap of position relationship between a micro lens array and 
a photo detector^ between a micro lens array and a photo detector. 

When the object image configuring meaxis conducts the shift amount 
calculating process, it may fiirther conduct a. projection process deriving process for 
15 obtaining a conversion equation indicating the geometric projection process employed 
in the reduced image estimating process based on the shift amoxmt obtained in the shift 
amoimt calculating process. 

Since such as the relative position and the focal length between micro lenses 
imaging each of the reduced object images are already known, when the shift amount 
20 indicating the gap of relative positions between the reduced object images is calculated, 
the distance between an object as a photographic subject and the micro lenses can be 
calculated. Therefore, based on such distance calculated as above and the known 
property of the camera, the conversion equation indicating the geometric projection 
process can be calculated. 
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Nioreover, when there is a misalignment between the micro lens array and the 
photo detector, and if the distance between the object and the micro lens army is 
calculated based on the shift amount calculated through correlation calculation of the 
reduced object images (in other words, to calculate the distance to the object based on 
5 the actual image photographed by a compoxind-eye camera), such distance may be 
considered the misalignment through the process such as averaging the distances to the 
object calculated for each of the reduced object images, and therefore^ the conversion 
equation indicating the geometric projection process calculated in accordance with such 
distance (a distance calculated based on a shift amount^ becomes as having the 

lO misalignment compensated. 

Thus, it is unnecessary to set the distance between the camera (micro lens 
array^ and the object in each case, and even if such distance is unknown (inconstant}, or, 
even if there is a misalignment between the micro lens array and the photo detector, an 
object image of high-resolution can be obtained. 

1 5 And in a similar way, when the object image configuring means conducts the 

shift amoimt calculating process, the generating process of initial object image 
conducted by the object image corLfiguiing means may generate an image data of a 
single object image by arranging a plirrality of the reduced object images onto a same 
area based on the shift amo\mt obtained in the shift amoimt calculating process, then 

20 may generate an initial image data of the single object image by interpolating blank 
pixels relative to the image data. 

Thus, relative to the initial image data, the misalignment in hardware is 
compensated, and the object image configuring that fuses together the compensation of 
the misalignment and the digital super-resolution processing (processing of the iterative 
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back projection method]) can be executed smoothly. 

In addition, the jnresent invention can be regarded as an image configuring 
method equivalent to the processing executed by the image conQguring apparatus. 

In other words, the above-mentioned image configuring method configures a 
5 single object image based on a image data of a plurality of reduced object images 
obtained by photographing an object using a compound-eye camera that focuses a 
plurality of reduced object images on a photo detector through micro lens array having a 
plurality of micro lenses arrayed therein, and is characterized by conducting: a 
generating process of initial object image for generating an initial image data of a single 

10 object image based on the image data of a plurality of reduced object images; a reduced 
image estimating process for estimating an estimated image of each of the reduced 
object images from a image data of a provided single object image based on a geometric 
projection process; and an object image updating process for updating an image data of 
the single object image provided in the reduced image estimating process by projecting 

15 a difference between the estimated image of each reduced object images and each of the 
reduced object images in an inverse process of the geometric projection process; and an 
iterative control process for firstly giving the initial image data to the reduced image 
estimating process as an initial value of the image data of the single object image, and 
then repetitively conducting the reduced image estimating process as well as the object 

20 image updating process until the diSerence satisfying the predetermined condition, then 
outputting the image data of the single object image at the time of the diflference 
satisfying the predetermined condition as a final image data of the object image. 
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BRTEK DESCMTI^IOTSr OK XBDE DRAWINGS 

Fig. 1 is a block diagram sliov\diig a. general structure of an image configuring 
apparatus X according to an embodiment of the present invention; 

Fig. 2 is a flow chart showing processing steps of the first example of image 
5 processing conducted by Image forming apparatus 

Figs. 3 is a fiow chart showing processing steps of the second example of image 
processing conducted by Image forming apparatus 

Figs. 4 (a) (b) and (c) respectively show an example of an object image 
configured by the first example of image processing; 
10 Figs. 5 Qa) and Ct>) respectively show an example of an object image configured 

by the second example of image processing; 

Figs. 6 (a) and (b) respectively show a correlation function between reduced 
object images; 

Fig. 7 shows an arrangement of a plurality of reduced object imeiges on a same 
15 area in a general manner. 

Figs. 8 {sl) and Qy} respectively show an optical system of a single-eye camera 
and an optical system of a compoimd-eye camera in a general manner. 

PEXAIl.EP DESCMPXION OF THE PREFERRED EMBOrHMENTS 

20 According to the present Invention, an object image of higher-resolution than 

the one of a high-resolution image obtained by the pixel rearranging method can be 
obtained, by generating an initial image data of a single object image based on a image 
data of a plurality of reduced object images obtained by a compound-eye camera, and 
then, as setting said initial image data as an initial value^ repeatedly conducting the 
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estimating processing of an estimated image of each, of tlie reduced object images from 
an image data of a single object based oxi a geometric projection process, as well as the 
updating processing of the image data of the single object image by projecting the 
difference between the estimated image of each of the reduced object images and each 
5 of the reduced object images in an inverse process of the geometric projection process, 
imtil the di£ference satis:fying with the predetermined condition. 

A.dditionally, this invention achieves to obtain an object image of 
high-resolution even when a distance between a camera (micro lens array^ and an object 
is imknown Cii^^oi^stantX or even when there is a misalignment between a micro lens 

lO array and a photo detector, by calculating a shift amount related to the gap of relative 
positions between the reduced object images through correlation calculation between 
the reduced object images using the image data of a plurality of the reduced object 
images, and then calculating a conversion equation indicating the geometric projection 
process based on the shiO: amount. 

15 With embodiments of the present invention described hereinafter with 

reference to the accompanying drawings, it is to be understood that the invention is not 
limited to those precise embodiments, and that various changes and modifications may 
be effected therein by one skilled in the art without departing fiom the scope or spirit of 
the invention as defined in the appended claims. 

20 Referring now to the block diagram illustrated in Fig. 1, the structure of an 

image configuring apparatus X according to an embodiment of the present invention is 
described. 

Image configuring apparatus !X is a general calculator such as personal 
computers, and comprises: a CPUl for executing various calculation processing, a 
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RAM2 as a main memory for developing a program when CPUl conducts an image 
processing program, a ROM3 in which a basic program forming a part of the operating 
system of Image configuring apparatus X is recorded, and a recording member 4 as a 
recording means for such as hard disks in which image processing programs as well as 
5 various data are recorded. Furthermore, thougih not shown in Pig. 1 , Image configuring 
appamtus X also comprises other structural elements a general calculator comprises. 

Image configuring apparatus JC is connected to a compoimd>eye camera. lO 
having the structure indicated in Fig. 8 Qd) through a A/D converter 20, and inputs an 
image data converted fi-om a plurality of reduced object images obtained by 
10 photographing an object by Compound-eye camera ID into a digital signal by A/D 
converter 20, thereby recording the image data into Recording member 4- Cone example 
of the reduced image recording means^. 

^Moreover, Image configuring appamtus 'X reads the image data of a plurality of 
the reduced object images &om Recording member 4 and configures a single object 
15 image bsised on such read image data (reduced object image^, then outputs (stores^ the 
image data into Recording member A- and, at the same time, as a graphic image, outputs 
into a display means such as OR.T displays not shown or into an image forming means 
such as printers. 

As shown in Fig. 8, Compoxmd-eye camera lO focuses a plurality of reduced 
20 object images on a photo detector 81 through a micro lens array 82 in which a plurality 
of micro lenses are arranged, thereby outputting a signal (^analog signal^ of each of the 
reduced object images. 

[The first example of image processing] 
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Next, as referring to the flow chart illustrated, in Fig. 2, a step for the first 
exazople of image processing conducted by Image configuring apparatus X! is explained. 
The present image processing is for configuring a single object image beised on an 
image data of a plurality of the reduced object images and outputting said image data, 
5 executed by CPUl (one example of the object image configuring means}' in Image 
configuring apparatus X! in accordance -with an image processing program set 
preliminarily. The folloiving SIO, S20 and so on indicate the number of processing 
steps. Prior to the execution of the present image processing, an Image data of a 
plurality of reduced object images obtained by photographing a pbotogenic subject 
lO (hereinafter referred to as an ^'object") using Compound-eye camera lO is assirmed to be 
recorded in Recording member 4 (one example of the reduced image recording means}. 

Here, it is assumed that there are no misalignment between Micro lens array 82 
and Pboto detector 81, and further, the distance between the object and Nlicro lens array 
82 is already known (including 'v^^en a result measured by some sort of methods is 
15 prescribed}. Also, the arranging intervals of each, of !Micro lenses 80, as well as the 
focal length are also Icnown. With these known information, based on known optical 
theories, by calculating the magnification ratio of the reduced object optical system 
from a distance to the object, and fiirther, clarifying the relationship between a pixel of a 
reduced object image and the object area, a conversion equation Tk, that indicates a 
20 geometric projection process from an actual image of the object to eacli of the reduce 
object image, and its inverse conversion equation Tjc"^ can be obtained. 

In the first example of image processing, such Tjc and Tic"^ obtained as above 
are assumed to be preliminarily provided. Here, the subscript **k*' indicates an 
identifying number (k^l ~ K.} of each of the reduced object images. Tk, for example. 
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can be indicated by such as functions or determinants of matrix, and Tk'^ can be their 
inverse conversion functions, inverse matrix, or pseudo inverses matrix. 
<Step S10> 

Firstly, reading an image data gk of a plurality of the reduced object images 
5 firom Recording merriber 4, then generating an initial image data f^*'^ (initial 
high-resolution image^ of a single object image based on the read image data gk C^ug 
example of the generating process of an initial object imaged . 

Here, an initial image data f^*^^ is obtained by applying image data of a 
plurality of the reduced object images to the following expression (1) which is based on 
lO an inverse conversion equation Tk'^. 
[Ivlathematical expression IJ 

=^±.T,-\g^:>*p ...(1) 

In Hxpression '^^ stands for a convolution arithmetic expression, and p is a parameter 
referred to as '^ack projection kemeP', both are to be defined so as to satisfy expression 
15 (5) described later. 
<Step S20> 

KText, estimating an image data gk^"-* of an estimated image of each of the 
reduced object images from provided image data f*^"-* (n=0, 1, 2, ...) of single object 
image 0>ased on a geometric projection process) through the following expression (2) 
20 which is based on the conversion equation Tk indicating a geometric projection process 
from an actual image of the object to each of the reduce object images (one example of 
the reduce image estimating process). 
[Ivlathematical expression 2] 



14 



Here, n stands for the mimber of times of a repetitive processing (iterative 
processing) described later, and provides the initial image data f^°^ obtained in SIO as an 
initial value (n=0) of Image data f^"^ of the single object image. In addition, h stands 
5 fbr a known blurring process (liereinafber, as well) in each, of reduced object optical 
system including each of Micro lenses 80 and Photo detector 8 1 (each element 
configuring the array). For example, with respect to h, an object as a photographic 
subject and reduced object images may be ejcpressed by a matrix vector, and a point 
spread function in optical system may be employed as a matrix. When a point spread 
lO function is imknown, a Oaussian distribution function may be employed. 
<Step S30> 

l>Text, obtaining a difference indicator e^"^ as an evaluation indicator of the 
diSerence (gap) between an estimated image gt^"^ of each of the reduced object images 
estimated in S20 and each of the reduced object images gk. 
15 In concrete, for example, the above is obtained through the next expression (3). 

pviathematical expression 3] 

<Step S40> 

Next, determining whether or not the difference indicator e^"^ obtained in S30 
20 satisfies the predetermined condition (in short, whether the difference satisfies the 
predetermined condition). 

Here, whether the difference indicator e^"^ becomes less than or equal to the 
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prescribed threshold is the predetermined condition. In addition to this, for example, 
other predetenxiined conditions may be employed, sucb as setting the predetermined 
condition to whether or not an improved width, (a lowering width) &om the difference 
indicator e^"^ in the prior time becomes less than or equal to the prescribed threshold. 
5 <Step S50> 

hi S40, when it is determined that the difTerence indicator e^"^ does not satisfy 
the predetermined condition, (in short, the difiGsrence does not satisfy the predetermined 
condition), an image data f^"^ of the single object image is updated by projecting (back 
projecting) the difTerence (gk-gk^"^) between an estimated image and a reduced object 

10 image in a inverse process of the geometric projection process based on the inverse 
conversion equation Tk~^ (one example of the object image updating process). 

Here, when the image data of the single object image after updating is assumed 
to be f^"**\ for example, the updating process is conducted through the following 
expression (4). 

1 5 [Nf athematical expression. 4] 

Here, in order to converge the difiference e^"^ by repetitively conducting S20 (in 
short, executing Expression (2)) and S50 (in short, executing Expression (4)), it has 
been understood that p should be prescribed so as to satisfy the following expression 
20 (5). 

[IVlathematical expression 5J 
\\S-h*J?\\^<l -..(5) 

Here, 6 stands for a delta function. In the processing S50, p should be 
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prescribed so as to satis:fy Expression (5). 

After conducting the above updating processing of the image data f^*^^ (S50), 
stepping back to S20, and repetitively conducting the above-mentioned processing with 
respect to the image data f^"^ of after updating. 
5 This provides the initial image data f^°^ obtained in SI O in S20 (the reduced 

image estimating process) as an initial value of the image data f^"^ of the single object 
image, and then repetitively conducts the processing S20 to S50 (the reduced image 
estimating process and the object image updating process) imtU the difierence e^"^ 
satisfies the predetermined condition. 
10 <Step S60> 

When the difierence e^"^ was determined to satisfy the predetermined condition 
in S40, the image data f^"^ of the single object image of the moment is output (stored) 
into R.ecording member 4 as an image data of a final object image, and at the same time 
into SL display means such as CRT displa3^ not shown as a graphic image, then sftsr that, 
1 5 the present image processing is ended. 

The processing in SBO^, S40 and S60 here are the example of processing of the 
iterative control process. 

Illustrated in Fig. 4 is one example of the image obtained by the present image 
processing. 

2.0 Fig. 4 (a) shows the reduced object image. Fig. 4 (hi) shows the object image 

obtained through the pixel rearranging method based on the reduced object image in Fig. 
4 (a), and Fig. 4 (c) shows the object image obtained by the present image processing 
based on the reduced object image in Fig. 4 (a). 

The number of aiTa3rs in IVHcro lens array 82 is 5 x 5, and each of the reduced 
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object images are formed in 80 x SO pixels. 

Tlie pixel rLumber of tbe object images (b) and (c) configured through the pixel 
rearranging raethod. or the present image processing are equivalent to the total pixel 
number (400 x 400 pixels) of the original reduced object images. 
5 From the comparison between Figs. 4 (h) and (c), it can be assured that the 

present image processing realizes production of a high-resolxition image. Particularly, 
sbaxpening e:Sect in the edge area as a high-firequency component of the image should 
be noted. 

[The second example of image processing] 
lO In the first example of image processing, it is assximed that there are no 

misalignment between Ivlicro lens array 82 and Photo detector 81, and further, the 

distance between the object and Micro lens array 82 is already known. 

However, in a practical system, a misalignment between a micro lens array and 

a photo detector is o:6:en imavoidable in the current position adjustment technology. 
15 In addition, fixing the distance between Compoimd-eye camera and an object, 

as well as measuring and setting the distance every time the distance changes are not 

practical. In the iterative baclc projection method, the above concerns related to the 

di£ference amount are not considered. 

Hereinafter, as referring to the flow chart illustrated in Fig. 3, the second 
20 example of image processing, that is capable of compensating the misalignment 

between Micro lens array 82 and Piioto detector 8 1 as well as the change of the distance 

between the object and Micro lens eirray 82, is explained. 

The present image processing is same as the first example of image processing, 

except replacing S 1 to S4 described later with the generating processing of initial image 
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data. (SIO) described in the first example of image processing (Fig, 2). CPUl of Image 
processing apparatus (oug example of the object image coixGguring means) executes 
the image processing according to the image processing program preliminarily set also 
in the present image processing. The &>llowing SI, S2 and so on indicate the ntimber 
5 of processing steps. Prior to executing the present image processing, an image data of 
a plurality of reduced object images obtained by photographing a photogenic subject 
(hereinafter referred to as an 'object') using Compound-eye camera is assumed to have 
been recorded in Recording member 4 (one example of the reduced image recording 
means). 
lO <:Step Sl> 

Firstly, obtaining a shift amount in regard to a gap of relative positions between 
the reduced object images through correlation calculation between the reduced object 
images by using the image data of a plurality of the reduced object images (one example 
of the shift amovint calculating process). 
15 A. gap of relative positions between the reduced object images is gen^ated 

caused by the parallax between each optical system (micro lens). 

The details on the present processing are explained hereinafter. 
Firstly, the relative shift amount of the reduced object image is hypothetically 
limited to the parallel shift for simplicity. 
20 Normalized cross correlation function Rc (x, y) of two of two variable 

functions f (x, y) and g(x, y) is generally defined through the following expression (6). 
[Mathematical expression 6] 
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f r f(.:x',y")s(.x'-x,y'~y')d^'cly' 

Since eacli of the reduced ol^ject image bas an identical shape, when two 
vaiial^le functions f([x, >0 and g(x, >0 assumed being identical, the correlation signal 
waveform Rc (x, y) becomes s3rEnmetric about a correiational peak Cx peak in Fig.6) as 
shown in a dotted line in Fig. 6Ca). In this case, the relationship of the following 
expression (7) is approved in between Rc (x, y) and x peak- Here, Gxpression C'7) is 
expressed in one dimension for simplicity. 
JlV4athematical expression 7] 



---(7) 

Consequently, when the correlational p>eak between the reduced object images 
is calculated by applying Bxpressions C^) and (7) to the reduced object images of 
Compound-eye camera lO, the relative position bertween the reduced object images can 
15 be figiu-ed out. 

Here, since the reduced object images are constituted from dispersed points 
that are sampled by Photo detector 81, the normalized cross correlation function 
becomes discrete, and therefore, a discrete correlation flmction Rd(xi, yj) defined in the 
following expression (8) is applied (identified by arrows in Fig. 6Ca)). A.dditlonally, 
20 the sampled shift positions are indicated as {Xi:i=0,l,2 yj:j=0, 1,2, M)-. 
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pVfathematical expression 8] 



2 2 f^^k^yt^sOCk -^i^yi -yj^ 



Furdiermore, in. order to obtain the shift amomit indicating the correlational 
peak of* the correlation function R.d(xi, yj), the correlate on ftmction Rc'(x, y) identified in 
fiill line in Fig. 6(a) is obtained by conducting linear interpolation according to the 
Ibllov^ng expression (9). £x] stands for the integer part of x. 
[Mathematical expression 9] 



lO --'C9) 

Here, as sliown in Fig. 6(a), since the correlation function Rc' (x, y) obtained 
through Expression C^) is what a discrete correlation function is linear interpolated, it is 
impossible to directiy identify an accurate x peak fir>m the correlation function Rc' (x, 

y>- 

15 There, the correlation function Rc' (x, y) becoming symmetric about x_peak 

since the reduced object images are identical can be utilized. In other words, by 
applying the Rc' (x, y) to £spression (7), x peak is figured oxit based on the shift 
amount that bisects the area obtained by integrating Rc' (x, y) within the prescribed 
range, as shown in Fig. 6Cb). 
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The above enables fig;uring out of tlie relative position between each of t±ie 
reduced object images. 

By repetitively executing the steps similar to the above with respect to every 
reduced otgect image, the relative position of every other reduce object image 
5 Ca^igiunent gap^ in regard to a reference reduced object image is jQgured out. 

With this amoimt of alignment gap» the corresponding relationship between the 
reduced object images including misalignment and the reconstructed images, in short, 
the shift amount (moving amount) of each, pixel necessary for configuring a single 
object image, can be obtained. 
10 As shown in Fig. 6(hy, by providing a prescribed threshold value T, integration 

of only the data that overtalces the threshold value may be conducted when figuring out 
the X peak. Such provision of the threshold value T enables calculating the relative 
position (in short, shift amount) by using the data near the correlational peak between 
the reduced object images, thereby figuring out more accurate relative position 
1 5 C^Ugnment gap). 

Additionally, the distance between the object and the apparatus can be derived 
from the known parameters of hardware, such as, the shift amount, the lens interval of 
the micro lens array, and the focal length. 
<Step S2> 

20 Next, based on the shift amount obtained in SI (shift amount calculating 

process), the conversion equation Tk indicating the geometric projection process 
employed in S20 (one example of the reduced image estimating process), as well as its 
inverse conversion equation Xk"* are obtained (one example of the projection process 
deriving process). 
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Since such as the relative position between the micro lenses photographing 
each of the reduced object images and the focal length of the micro lenses are known, 
when the shift amount indicating the gap of relative positions between the reduce object 
images is obtained, the c3i stance between the object as a photogenic subject and ;h4icro 
5 lenses 80 as well as the multiplication ratio of the optical system can be obtEiined. A. 
conversion equation Tk indicating the geometric projection process by a. theoretical 
equation of a known optical system can be obtained &om the above ratio and an area 
(extent) of a. photo detector per pixel. Furthermore, &om the conversion equation Tk, 
its inverse conversion equation Tk~' (such as, the inverse mattix and pseudo inverse 

lO matrix) can be obtained. 

Here, the distance to the object, under the condition that the interval between 
Micro lens array 82 and Photo detector 81 is ideally aligned, can be derived with 
accuracy by obtaining an object distance (distance to the object) figured out &om the 
shift amotmt per pixel for every reduced object image and then averaging these object 

15 distances. 

A.ccordingly, if there is a misalignment between IVlicro lens array 82 and Photo 
detector 81, as mentioned above, the misalignment is considered to a distance to the 
object obtained on the basis of the shift amotmt obtained througih correlation calculation 
of the reduce object image, and a conversion equation indicating the geometric 
20 projection process obtained on the basis of such distance achieves compensation of the 
misalignment. Here, when the shape of each pixel of Photo detector 81 is known, an 
optional fill factor ratio of the photo detector can be employed. 

Consequently, it is unnecessary to set up the distance between Compound-eye 
camera lO (micro lens array) and the object in each case, and even when the distance is 
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unknown (inconstant), or ttiere is a misalignment between the micro lens array and the 
photo detector, the conversion equation indicating the geometric projection process 
reflecting these can be derived, 
<Step S3> 

5 Next, generating an initial image data of a single object image of which the gap 

of a relative position is amended, by arranging a plxurality of the reduced object images 
on a same Eirea based on the shift amount obtained in SI C^hift amount calculating 
process). 

Fig- 7 shows arrangement of a plurality of the reduced object images on a same 
lO area in a general mamier. QO, Ql and Q2 respectively indicate the reduced images 

having 3x3 pixels focused on Photo detector 81, and IVll is the area where a single 

target image is configured. The area Nil is reserved, for example, in a part of the area 

in IRAJM. 2, and its nutmber of pixels is assumed to be the same as the number of pixels 

in the entire photo detector. 
15 in the present processing, firstly, arranging the pixels of the reduce object 

images (ii^clicated at QO in Fig. 7), which are set as the standeird in SI, in accordance 

with the scale of in the area !N11, as indicated by arrows. 

Then, arranging the pixels of the other reduced object images (indicated as Ql 

and Q2 in Fig. 7) into the area Ml based on the shift amount calculated in SI. 
20 Consequently, a single object image of high-resolution, in which the gap of 

relative position between the reduced images are amended and arranged on the area Ml 

is configured. 

<Step S4> 

Next, conducting the interpolation processing of the blank pixels in regard to 
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the image data of the object bnage obtained in S3. Here, S3 and S4 are the example of 
the generating process of initial object image. 

If there are no blank pixels In the image data of a single object image obtained 
in S3» this image data can directly be the initial image data. However, the single object 
5 image obtained in S3 is the image in which the pixels of each reduce<l object image are 
merely arranged into the expanded same area, and therefore, it might include a blank 
pixel, depending on the object distance between the object and Micro lens array 82, or if 
there is the misalignment. Using such image including a blank pixel as an initial 
image data, blanlc pixels may become noise factors and such noise signal is emphasized 
lO in the iterative back projection process described later, so it is not appropriate as the 
initial image data. 

Additionally, when there is the misalignment, if the initial image data is 
generated through application of the inverse conversion equation Tk~* obtained without 
consideration of such misalignment like a conventional iterative back projection method, 

15 there generates a difference in corresponding relationship between a reduced object 
image and an estimated reconstructed image. This difference indicates that there is a 
drOerence between the actual object information and the reconstructed image. The 
accuracy of the initial image (the amount of the difTerence} afiTects the speed (time) of 
the iterative processing as well as the image quality of the high -resolution image finally 

20 obtained. Therefore, it is not appropriate to generate the initial image data by applying 
the inverse conversion equation Tic'' obtained without consideration of the 
misalignment. 

Therefore, when there is a blank pixel, intezpolation is conducted by using 
pixel values of neighboring pixels. In such case, when it is asstmied that the pixel 
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VEilvxe changes linearly in the horizontal or the perpendicnlar directions in a blaxLk pixel, 
and if Ndci unit of the blank pixels exist successively, the pixel value Pdej(i) interpolating 
the i til blaxiJc pixel can be calculated through the fblloAving expression (lO^ for example. 
Here, P 1 and P2 are pixel values of the pixels in vicinity of a blank pixel. 
[IMathematical expression 10] 

■••CIO) 

An average value of values obtained by applying Expression (lO) in the 
horizontal or the perpendicular directions in respect to the target blank pixel is the pixel 
value of the blank pixel. 

With the above, blank pixels are eliminated and an image which is close to an 
actual object image with high resolution can be obtained. 

Hereinafter, executing the processing in S2Q to S60 mentioned above, by usii^ 
the image data of an object image after interpolating the blEink pixel obtained in S4 as 
the initial image data, and further, by employing the conversion eqimtion Tic indicating 
the geometric projection process as well as its inverse conversion equation T^"' obtained 
inS2. 

With the above processing, it is no more necessary to prescribe a distance 
between Compound-eye camera lO (micro lens array) and the object in each case, and 
even when such distance is unknown (inconstant), or there is a misalignment between 
the micro lens array and the photo detector, an object image of high-resolution can be 
obtained. 
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One example of the image obtained by the present image processing is shown 

in Fig. 5- 

Fig. 5 (sC) shows an object image obtained through the pixel rearranging 
method based on a plurality of reduce object images obtained by Compound-eye camera 
5 10, and Fig. 5 (b) shows an object image obtained by the present image processing 
based on the reduce object images as well. 

The imaging condition of the example in Figs. 5 (a) and (bt) are as follows. 
The focal length for each micro lens is 1 .3 millimeter, the lens diameter is 0.5 
millimeter, and the photographic object is placed and photographed in the distance of 
10 520 millimeter from the micro lens. The obtained image is consisted of 5x5 (TSr=5) 
unit of the reduced object images, and each reduced object image has 160 x 160 pixels, 
and a color CCI3 using a RGB filter based on the Bayer array (handling the information 
of a. set of channels Q and B in 2 x 2 pixels} is used as a light receiving member. 
Consequently, the information of each channel CB^ and B is 80 x SO pixels per reduced 
15 object optical system. 

Hach of the object images in Figs. S(aL) and (ht) are consisted of AOO x AOO pixels. 
When comparing Figs. 5Ca) and G>)» though unevenness in the degree of 
high-resoluttions can be seen, the eyes and the bill of the hat can be recognized clearly. 

As a result of the above, from the present image processing example 2, it can be 
20 understood that an object image of high-resolution can be obtained, even when the 
distance between a camera Cmicro lens array) and an object is unknown (inconstant), or 
there is a misalignment between a micro lens array and a photo detector. 
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